High speed stereoscopic pavement surface scanning system and method

ABSTRACT

Disclosed is a mobile pavement surface scanning system for detecting pavement distress. In an embodiment the system comprises one or more light sources mounted on the mobile vehicle for illuminating a pavement, one or more stereoscopic image capturing devices mounted on the vehicle for capturing sequential images of an illuminated pavement surface, and a plurality of positioning sensors mounted on the mobile vehicle, the positioning sensors adapted to encode movement of the mobile vehicle and provide a synchronization signal for the sequential images captured by the one or more stereoscopic image capture devices. One or more computer processors are adapted to synchronize the intensity image pairs captured by each camera in the one or more stereoscopic image capturing devices, perform a 3D reconstruction of the pavement from the intensity image pairs using stereoscopic principles, generate a depth image and an intensity image pair from the 3D reconstruction, and process at least one of the depth image and the intensity image utilizing one or more distress detection modules to detect a type of pavement distress.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 14/996,803 filed on Jan. 15, 2016, which is hereby incorporatedby reference in its entirety.

FIELD

This disclosure relates broadly to surface digitization systems andmethods for accurate detection and assessment of pavement profiles andthree dimensional (3D) surfaces for the purposes of detecting andmeasuring pavement distresses.

BACKGROUND

An accurate assessment and identification of road pavement surfaces isrequired for timely maintenance of roads (pavements). Pavements developmany different modes of distresses over time, including but not limitedto cracking, rutting, faulting, ponding, spalling and ravelling (i.e.on-going separation of aggregate particles in a pavement). The conditionof the pavement can be determined by assessing the type, extent,relative and absolute location, and severity of each of these differenttypes of distresses, and remedial measures can be applied to fix theseproblems. In addition, it is also important to measure the roughness andtexture of pavements periodically. Textures helps to measure the skidresistance, and roughness measures the level of traveler comfort andimpact on fuel efficiency. Pavement surface conditions are usuallyassessed using survey vehicles which continually collect pavementsurface data as they travel along their designated routes. A number ofpavement condition assessment systems have been built in the past fourdecades. These systems use different sensors to digitize the roadsurface and roughly fall under one of the following two categories:

-   -   (1) Imaging systems, which use a camera or sets of cameras and        lighting systems to record a view of the pavement surface. These        systems usually use high resolution line scan cameras for        accurate imaging. The individual lines scanned by the camera are        stitched after some distance to get a two-dimensional image of        the area scanned. They capture an entire area of the lane in        which the survey vehicle is travelling in. Surface data captured        with these systems are usually used for distress detection.        However, these systems are two-dimensional (2D) as opposed to        three-dimensional (3D).    -   (2) Profiling systems, which use laser triangulation, ultrasound        or other time of flight sensors to record the elevation map of        the pavement surface. These systems do not measure the entire        surface of the road, but rather produce profiles at fixed        intervals along a fixed number of lines on the road. While these        systems are highly accurate and measure discrete points across        the surface of the road, these systems take discrete        measurements and therefore do not by their nature take images,        as the 2D imaging systems described above do.

The recorded road surface is then either assessed manually orautomatically according to various pavement assessment standards.

Stereoscopy is the extraction of three dimensional (3D) elevationinformation from digital images obtained by imaging devices such as CCDand CMOS cameras. By comparing information about a scene from twovantage points, 3D information can be extracted by examination of therelative position of objects in the two panels. This is similar thebiological process Stereopsis, a process by which the human brainperceives the 3D structure of an object using visual information fromtwo eyes.

In the simplest form of the technique, two cameras displacedhorizontally from one another are used to obtain two differing views ona scene. By comparing these two images, the relative depth informationcan be obtained, in the form of disparities, which are inverselyproportional to the differences in distance to the objects. To comparethe images, the two views must be superimposed in a stereoscopic deviceor process.

For a two camera stereoscopic 3D extraction technique, the followingsteps are performed:

-   -   (a) Image Rectification: Transformation matrix R_(rect)        transforms both the images to one common plane of comparison is        identified. The left camera image is rectified by applying        R_(rect) and the right camera image by applying R*R_(rect) to        all the pixels.    -   (b) Disparity Map generation: For each pixel on the left camera        image a matching pixel along the same scan line is identified on        the right camera image using a localized window based search        technique. For each pixel, p_(l)(x,y) in the left image, the        system and method identifies the matching pixel p_(r)(x+d,y) in        the right pixel where d is the pixel disparity.    -   (c) 3D reconstruction: At each point d_((x,y)) in the disparity        map, the system and method calculates the elevation z_((x,y)) by        triangulation.

Stereoscopy has been used for pavement quality assessment in U.S. Pat.No. 8,306,747. The system utilizes Ground Penetrating Radar (GPR) alongwith stereo area scan cameras to obtain high resolution images, and isnot designed for operation at highway speeds. The system also does notuse the image data directly for distress detection and measurement.

Techniques similar to multiple-camera stereoscopy like photometricstereoscopy has also been used in pavement assessment in Shalaby et al.(“Image Requirements for Three-Dimensional Measurements of PavementMacrotexture”, Journal of the Transportation Research Board, IssueVolume 2068/2008, ISSN 0361-1981.) However, the system uses aconventional camera with four single point light sources, and is notdesigned for high-speed operation. The technique is used to characterizepavement surface textures.

Stereoscopic imaging has also been used for inspection of objects on aconveyor belt using both individual photo-sensors (U.S. Pat. No.3,892,492) or using a line-scan camera (U.S. Pat. Nos. 6,166,393 and6,327,374). They e also specifically designed to identify defectiverapidly moving objects moving on a conveyor belt past a stationarysensor system rather from a moving platform for road pavementevaluation.

What is therefore needed is an improved system and method for pavementscanning that overcomes some of the disadvantages of the prior art.

SUMMARY

The present disclosure relates to a high speed pavement stereoscopicline scan imaging system and method capable of producing a stereoscopic3D image of the pavement surface using a stereoscopic image capturingapparatus, or any number of such devices and lighting source(s) foraccurate detection of pavement distresses, and assessment of thepavement surface quality. The present system and method can be appliedto capturing and assessment of any type of pavement or vehicle pathwaysurface, such as road pavements, bridge decks and airport runways andrailways.

In an aspect, there is provided a mobile pavement surface scanningsystem for detecting pavement distress, comprising: one or more lightsources mounted to a mobile vehicle for illuminating a pavement surface;one or more stereoscopic image capturing devices mounted to the mobilevehicle for capturing sequential images of the illuminated pavementsurface; a plurality of positioning sensors mounted to the mobilevehicle, the positioning sensors adapted to encode movement of themobile vehicle and provide a synchronization signal for the sequentialimages captured by the one or more stereoscopic image capture devices;and one or more computer processors configured to: synchronize thesequential images captured by each camera of the one or morestereoscopic image capturing devices; generate intensity image pairsfrom the synchronized sequential images; perform a 3D reconstruction ofthe illuminated pavement surface from the intensity image pairs usingstereoscopic principles; generate a depth image and an intensity imagepair from the 3D reconstruction; and process at least one of the depthimage and the intensity image utilizing one or more distress detectionmodules to detect a type of pavement distress.

In an embodiment, the one or more distress detection modules comprise acomputer vision module for detecting pavement distress utilizing atleast one of the depth image and an intensity image pair.

In another embodiment, the one or more distress detection modulesfurther comprise a machine learning module, and the computer visionmodule is adapted to generate a learning feed forward to the machinelearning module.

In another embodiment, the one or more distress detection modulescomprise a machine learning module for detecting pavement distressutilizing at least one of the depth image and an intensity image pair.

In another embodiment, the machine learning module includes a learningfeedback loop to enable the machine learing module to improve detectionof pavement distresses.

In another embodiment, the machine learning module comprises anartificial intelligence (AI) engine executing a learning algorithm todetect and classify distresses based on its iterative training.

In another embodiment, the machine learning module is adapted to providea feedback signal to dynamically change a parameter of a component onthe mobile vehicle for capturing the sequential images on theilluminated pavement surface, such as the one or more light sources, oneor more stereoscopic image capturing devices, and filters.

In another embodiment, the machine learning module is adapted to selecta type of image processing filter in dependence upon the type ofpavement distress being detected.

In another embodiment, the machine learning module is adapted tocategorize the type of pavement distress detected, and to store thegeo-reference for the detected pavement distress in real time as thesurvey data is stored.

In another aspect, there is provided a method of scanning a pavementsurface for detecting pavement distress, comprising: providing one ormore light sources mounted to a mobile vehicle for illuminating apavement surface; providing one or more stereoscopic image capturingdevices mounted to the mobile vehicle for capturing sequential images ofthe illuminated pavement surface; providing a plurality of positioningsensors mounted to the mobile vehicle, the positioning sensors adaptedto encode movement of the mobile vehicle and provide a synchronizationsignal for the sequential images captured by the one or morestereoscopic image capture devices; and providing one or more computerprocessors configured to: synchronize the sequential images captured byeach camera of the one or more stereoscopic image capturing devices;generate intensity image pairs from the synchronized sequential images;perform a 3D reconstruction of the illuminated pavement surface from theintensity image pairs using stereoscopic principles; generate a depthimage and an intensity image pair from the 3D reconstruction; andprocess at least one of the depth image and the intensity imageutilizing one or more distress detection modules to detect a type ofpavement distress, as described above for the corresponding system.

Further features will be evident from the following description ofpreferred embodiments. In this respect, before explaining at least oneembodiment of the invention in detail, it is to be understood that theinvention is not limited in its application to the details ofconstruction and to the arrangements of the components set forth in thefollowing description or illustrated in the drawings. The invention iscapable of other embodiments and of being practiced and carried out invarious ways and equivalents to the embodiments. Also, it is to beunderstood that the phraseology and terminology employed herein are forthe purpose of description and should not be regarded as limiting.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows one possible configuration of the scanning system mountedon the survey vehicle. The system shown has two pairs of stereoscopicline-scan cameras and two light sources in accordance with anillustrative embodiment.

FIG. 2 is one possible configuration of a stereoscopic line-scan camerapair and a light source shown together in accordance with anillustrative embodiment.

FIG. 3 is a schematic block diagram of the scanning system in accordancewith an illustrative embodiment.

FIG. 4A is a schematic block diagram the data capture scheme used forthe scanning system in accordance with an illustrative embodiment.

FIG. 4B. is a schematic block diagram of the image processing schemeused for the scanning system in accordance with an illustrativeembodiment.

FIG. 4C. is a schematic block diagram of the data post-processing schemeused for the scanning system in accordance with an illustrativeembodiment.

FIG. 5 shows sample grayscale images of a pavement surface captured byleft and right cameras of a stereoscopic image capturing device inaccordance with an illustrative embodiment.

FIG. 6 shows a representative 3D image of the pavement surface obtainedusing the images shown in FIG. 5 in accordance with an illustrativeembodiment.

FIG. 7 shows a schematic block diagram of a process for utilizingintensity and depth images for detection, classification and analysis ofpavement distresses.

DETAILED DESCRIPTION OF THE INVENTION

As noted above, the present disclosure relates to a system and methodfor collecting a high resolution 3D image of the pavement surface athigh speed, and utilizing the captured 3D image for detection,classification and analysis of pavement distresses. The purpose of thesystem and method is to collect information that allows a more accuratemeasurement of various different modes of distress that have formed on aroad pavement surface. These measurements can then be used to manuallyor automatically assess road condition, such as cracking, roughness,smoothness, rutting and both micro and macro surface texture.

In an embodiment, with reference to FIGS. 1 to 4B, the proposed systemis mounted to a survey vehicle, and comprises a number of elements: (1)A number of high brightness illumination units, suitably two LED sources130A and 130B (in an embodiment, these may be of blue wavelength rangingfrom about 450 nm to 495 nm, and more preferably around 480 nm, butother colors and corresponding wavelengths may be used); (2) A number,suitably two, of stereoscopic image capture devices 104A and 104B whichmay include pairs of high speed line scan cameras 120A & 120B, 120C &120D, and frame grabbers 150A and 150B with each of the camerasexternally fitted with an optical filter 103A, 103B; (3) A combinationof wheel-encoder 105A, GPS 105B and IMU 105C mounted to the vehicleallowing movement detection; and (4) A data-storage 510 and processing520 means.

In an embodiment, the light sources 130A, 130B used to illuminate anarea of interest are adapted to receive a trigger pulse to synchronizethe output of the light sources 130A, 130B with the image capturingdevice. The intensity of the light output by the light sources 130A,130B may be modified depending on the amount of illumination a pavementsurface requires, in order to synchronize with the image capturingdevice and capture images with a suitable level of contrast. Theintensity of the light output by the light sources 130A, 130B may alsobe controlled by an exposure level sensor, such as an exposure levelmeter built into the camera providing a feedback signal. The camera lensaperture and the sensitivity of the camera image sensor may also becontrolled in order to obtain a proper level of exposure for a givenlighting condition.

The illumination system 130 may be one very powerful illumination sourcethat covers the entire width of a pavement surface of interest, ormultiple illumination sources comprising one or more LED sources 130A,130B that together cover the width of the pavement surface of interest.

When multiple sources are used, each source may be fitted together withan image capturing device, and housed together in a cabinet to beprotected from environmental damages, as shown by way of example inFIG. 1. One or more supplemental illumination sources positionedseparately from the cabinet may also be used as necessary in order toachieve proper illumination of the pavement surface. FIG. 1 shows anillustrative vehicle mounted system with two such cabinets 110A, 110Bwhich are mounted at the upper left corner and upper right corner of therear of the vehicle. As shown, these two units may be interconnected viacables through a ducted frame holding the two cabinets in position. Thetwo light sources 130A and 130B continuously illuminate the width of thepavement as the vehicle travels forward, in order to allow the one ormore stereoscopic image capture devices to record a sequence of pavementsurface images.

When multiple sources are used, a part of the width of the pavementilluminated by one source may overlap with the width illuminated by theothers as shown in FIG. 1. In FIG. 1 coverage width 140 is obtained bycoverage width 140A from a first light source 130A which partiallyoverlaps with coverage width 140B from a second light source 130B insidethe second cabinet 110B.

In an embodiment, the orientation of the light source 110 with respectto the pavement surface is determined by the cabinet. Inside thecabinet, the light source is placed with no rotation, with the beamparallel to one of the long faces of the cabinet as shown in FIG. 1. Thelight sources 130A, 130B may also be positioned at appropriate anglesand distances relative to each other in order to provide optimallighting conditions for obtaining a sufficiently high contrast image ofthe pavement surface features.

The image capturing system 104, may be one wide-angle stereoscopic imagecapturing device or multiple medium-angle or narrow-angle devices thatcapture the width of the pavement. A stereoscopic image capturing device104A consists of at least two cameras, left camera 120A and right camera120B. Both the left and right cameras capture almost the same width ofthe pavement 140A and 140B, as shown in FIG. 1 and FIG. 2, which formsthe basis of 3D depth (range) estimation using stereoscopic principles.Each camera may be a single integrated unit or a separate high speedline scan camera 120 and frame grabber 150 A and 150B.

Depending on the width 140 of the pavement surface to be captured andthe width 140A, 140B that a single stereoscopic pair can capture,multiple similar pairs may be used as shown in FIG. 1. Similar to theillumination system, when multiple image capturing devices are used, thewidth of the pavement captured by one stereoscopic pair may overlap withthe width captured by the others as shown in FIG. 1.

Each of the cameras in a stereoscopic camera pair may be fitted with anoptical filter or lens filter 103A and 103B externally or internally toovercome the environmental challenges like abnormal sunlight conditionor wet pavements.

FIG. 4A shows one possible configuration of a Data Capturing System. Theimage capturing system with two high speed stereoscopic line scan camerapairs 104A and 104B, in combination with optical filters that arematched to the wavelength of the light source, 103A and 103B, capturesthe pavement surface at high resolution, using frame grabber cards 150Aand 150B. The illumination system with two LED light sources 130A and130B illuminates the pavement surface.

A combination of a Global Positioning System (GPS) 105A, InertialMeasurement Unit (IMU) 105B and Wheel Encoder 105C, collectivelyreferred to as Distance Measurement Instruments (DMI) 105, detects themovement of the system as shown in FIG. 3. The individual sensors areplaced at different locations inside the survey vehicle. Together, theycapture any movement of the survey vehicle such as longitudinal distancetravelled, velocity in the direction of travel and angle of tiltrelative to pavement surface. DMI also produces synchronization signals201 based on distance travelled by the survey vehicle which is used totrigger the stereoscopic cameras for synchronized data captureindependent of the vehicle velocity as shown in FIG. 4B. DMI may alsoproduce the synchronization signals based on the time elapsed.

The movement data from the IMU is used to augment the data captured bythe image capturing devices to correct for pavement abnormalities andobtain more accurate 3D estimates. For example, if the vehicle istravelling over an uneven surface or stretch of banked pavement which isangled to one side or when the vehicle bounces, the IMU data is used toaccount for the movement of the system relative to the pavement surface.

As the survey vehicle travels forward, the image capturing devices aretriggered at equal distance or time intervals, in rapid succession, bythe DMI. In an embodiment, this trigger pulse may be generated using anencoder or vehicle speed sensor 105C, connected to the drive train ordirectly to the wheel. At each pulse, the individual cameras of astereoscopic pair capture a line of pavement surface illuminated by theillumination source. The captured lines are then digitized into a lineof grayscale intensities using the frame grabber card. The frame grabbercaptures a fixed number of such lines and stitches them together oneline after another to form a two dimensional (2D) image.

In this illustrative embodiment using a pair stereoscopic cameras, theresult is a set of four, time or distance synchronized, 2D intensityimages containing image intensity data. The intensity images captured bythe left and right cameras of one of the two stereoscopic pairs of asample system are shown in FIG. 5.

At this stage, the images are processed and saved as shown in FIG. 4B.Image processing comprises of external artifact removal 501, imagerectification 502, disparity estimation 503, 3D depth (range) estimation504, image stitching 505, and image compression 506. Image processing isperformed on-board 520, as the vehicle travels. Alternatively thesesteps can be done in a post-processing stage. A wireless communicationsmodule 530 may also transmit survey data as a live stream feed to aremote location for storage and processing.

As shown in FIG. 4B, the first step in image processing is to reduce theeffect of sunlight and shadows within the images. Initially, the opticalfilters on the stereoscopic cameras reduce the effects of sunlight.However to obtain good contrast images with accurate gradient estimates,further reduction of the effects of sunlight is often necessary. Torectify this problem, an ancillary image of the surface can be takenwith no artificial lighting, only sunlight. This image with onlysunlight illuminating the surface is then used to remove the effect ofsunlight in the other images collected by the system. This is performedafter each of the images has been aligned, as described previously. Bysubtracting the sunlight only image from the original images usingdigital processing, sunlight free images can be produced. This techniquealso removes the effect of imaging sensor DC bias. Alternatively, if anancillary image without artificial lighting cannot be taken, this stepmay be replaced with simple contrast normalization techniques 501 whicheffectively spread out the most frequent intensity value.

Once the external artifacts have been removed from the images, thetechnique of stereoscopy is applied to the data. This produces the 3Delevation at each point on the pavement surface. The preferred techniqueuses images from two individual cameras of the stereo pair and for eachpoint on the pavement, identifies the corresponding pixel on both theimages and estimates the 3D elevation as a factor of relative pixeldistance between the matching pixels. The stereo camera pairs arecalibrated and the focal length (f), principal centers (P) of theindividual cameras and the relative rotation (R) and Translation (T)between the two cameras are known.

The following steps are performed:

(a) The first step is Image Rectification 502. The system and methodidentifies a common R_(rect) matrix that when applied will transform theleft and right images to a common plane where they can be compared pixelto pixel. The system and method determines this R_(rect) matrix usingthe Translation vector (T).

$e_{1} = {{\frac{T}{T}\mspace{14mu} e_{2}} = {\frac{1}{\sqrt{T_{x}^{2} + T_{y}^{2}}}\left\lbrack {{- T_{y}},T_{x},0} \right\rbrack}^{\prime}}$${e_{3} = {{e_{1} \times e_{2}\mspace{14mu} R_{rect}} = \begin{bmatrix}e_{1}^{\prime} \\e_{2}^{\prime} \\e_{3}^{\prime}\end{bmatrix}}};$

The system and method rectifies the left image by applying the R_(rect)matrix to each pixel in the image. For each pixel, Pi, the system andmethod computes R_(rect)*p_(l). Similarly the system and methodrectifies the right image by applying R*R_(rect) to each pixel. For eachpixel, p_(r), the system and method computes R*R_(rect)*p_(r). Thistransforms both the images to one common plane for easy comparison.

(b) The next step is to generate a Disparity Map 503. For each pixel inthe left image, the system and method identifies a matching pixel in theright image. Since the images are rectified, the search space toidentify the matching pixel is limited to the corresponding scan line.The system and method uses a localized window based correlationtechnique to identify the matching pixels. For each pixel, p_(l)(x,y) inthe left image, the system and method identifies the matching pixelp_(r)(x+d,y) in the right pixel where d is the pixel disparity.

(c) The final step is 3D reconstruction 504. At each point d_((x,y)) inthe disparity map the system and method calculates the elevationZ_((x,y)) by triangulation.

$z_{({x,y})} = \frac{T_{x}*f}{d_{({x,y})}}$

The 3D pavement profile, obtained using the disparity image which isobtained using the grayscale images shown in FIG. 5, is shown in FIG. 7.

Once the 3D range maps are obtained from the stereo pairs, at 505, thesystem and method stitches the range maps obtained by the stereo pairsto obtain one 3D range map for the entire region of interest.

After image capturing, stereoscopic 3D reconstruction and imagestitching, the images obtained are contrast normalized intensity imagescontaining image intensity data (which may be gray scale), and 3Delevation/depth range images which are combined into a stereoscopic 3Dimage containing image intensity data. This stereoscopic 3D image isviewable as a 3D image rendered on a 2D computer monitor or screen, orviewable in stereoscopic 3D with suitable 3D glasses. With appropriateformatting as may be necessary, the 3D image may also be viewed in avirtual 3D environment, using a commercially available stereoscopicvirtual reality viewer, for example. Such a virtual 3D viewingenvironment may render pavement distress features in the stereoscopic 3Dimage to be more readily noticeable, in comparison to a flattenedrendering of a 3D image on a 2D computer monitor or screen. Once such afeature is identified, the viewing angle of the 3D image may also bemaniputed to allow the pavement surface to be viewed from differentpoints of view.

A file compression 506 technique such as GeoTIF, JPEG encoding, ZIPencoding and LZW encoding is applied to minimize the sizes of thecombined stereoscopic 3D images, and save them to a data storage device510 on board.

Any or all of the steps involved in image processing stage can beperformed by one or multiple units of Central Processing Unit (CPU) 520Aor Graphics Processing unit (GPU) 520B as shown in FIG. 4C.

At the post processing and extraction stage, the recorded data isretrieved from a data storage 510, decompressed 601, and then passed toa number of modules as shown in FIG. 4C.

The high resolution stereoscopic 3D image can be used to extract anumber of pavement features. Through the automatic identification andclassification of each of these features, an assessment of the roadsurface condition can be made 610. These include, but are not limitedto:

-   -   (1) Identification of surface cracking (both sealed and        unsealed) 604.    -   (2) Extraction of road roughness or smoothness 605.    -   (3) Identification of areas with low texture depth, which can be        due to asphalt bleeding or polishing 606.    -   (4) Identification of pot holes and rutting 607.    -   (5) Identification of areas where there is surface depression or        corrugation which can indicate areas of high moisture or        voiding.    -   (6) Extraction of Transverse Profile for rutting estimation 608.    -   (7) Surface comparison between scans, allowing detection of        surface change with time.    -   8) Identification and removal of spurious road targets such as        sticks and other debris, which can confuse crack detection        algorithms.    -   9) Identification of patches.    -   10) Identification of areas of water bleeding.

The 3D image can be used along with the contrast normalized intensityimages containing image intensity data to improve the distressdetection, especially, cracking 604. Cracks are identified both in thegradient and intensity images. Both the shape and intensity is then usedto classify the features as cracks, sealed cracks or other roadfeatures. The main advantages over using just the 3D image is theability to eliminate false targets, such as markings on the road. Anexample is an oil spill which is often incorrectly identified as acrack, as it will only appear within the intensity image, not the 3Drange images. It also improves the identification of other surfacefeatures that could lead to false positives, such as road markings,wheel marks, sticks and other road debris.

Another highly useful element of the system is the ability to identifysealed distresses like sealed cracks. Cracks are often sealed usingbitumen, which to a normal surface image camera still appear as a darkline within the image. With the stereoscopic 3D image estimationtechnique it is possible to detect the presence of the flat bitumensurface in contrast to the depression caused by an unsealed crack.

Modules may also employ Machine Learning techniques to detect thedistresses. The modules, instead of employing a series of mathematicalcalculations with hard-coded constants (heuristic methods), learn theshape and structure of the distresses from manually labelled historicaldata and try to predict the presence of distress on the capturedpavement image. Each distress type has unique characteristics and itrepeats wherever the distress appears again. Machine learning basedmodules are proven to be more accurate than heuristic method employingmethods for detecting objects in an image.

In the display module 603, the data produced can be displayed directlyto the user on the on-board monitor. The display module may display justthe intensity image or a combined intensity image and 3D elevationimage. According to the user preferences, the module may also displaythe detected distresses overlaid on the intensity image. The distressesdisplayed may be color-coded in different colors to indicate the levelof severity.

Now referring to FIG. 7, shown is a schematic block diagram of a systemand method for utilizing intensity and depth images for detection,classification and analysis of pavement distresses. First describing thesystem and method at a high level, FIG. 7 discloses an illustrativesystem and method in which left and right stereo images 712, 714 areacquired by a vehicle mounted system as described in the specification.The system and method then proceeds to a 3D reconstruction block 716.From this block, the system and method proceeds to generate both a depthimage 718 and an intensity image 720. The depth and intensity images718, 720 then undergo image processing 722. Processed images areprovided as an input to processing utilizing a computer vision module724, and a machine learning module 726. These modules 724, 726 thendetect, classify, and record details of the different types of pavementdistresses, including crack detection, surface roughness calculation605, surface texture analysis 606, rutting and pothole detection 607,transverse profile estimation 608, and other types of pavementdistresses.

These results are then processed for storage 728 for recall and possiblefurther analysis. The system and method will now be described in moredetail, with reference to FIG. 7 and earlier figures.

It will be understood that the system and method may be embodied on aprocessor, such as illustrated in FIG. 3, and the processor may beintegrated, whether wired or wirelessly, with image capture modules andsensors as provided on board a survey vehicle. It will be understoodthat any reference to a system or to a method as executed on the systemmay involve theses processors and modules as previously described.

As shown in FIG. 7, L & R stereo raw images 712, 714 are acquired by avehicle mounted system as described above. A 3D reconstruction 716 ofthe pavement is then performed utilizing stereoscopic principles, andextracting from the stereo raw images 712, 214 the information relatingto pavement distresses in order to generate a depth image 718, and anintensity image 720.

Capturing a stereoscopic L & R image pair of raw images 712, 714 enablesretrieval of a 3D representation of the scanned pavement surface withoutlosing any details. For example, several different types of distressesonly manifest themselves in term of height differences, without anyother really noticeable noticeable features. The 3D sensors of thepresent vehicle mounted system collect the necessary data to detectthese distresses in x, y and z dimensions. These 3D sensors of thevehicle mounted system, in combination with the detection andclassification processes as now described, allows extraction of highlyaccurate detailed features of pavement distresses, using 3D depthmeasurements in combination with imaging.

Once the depth image 718 and intensity image 720 are generated, andimage processing step 722 is utilized to ensure that the images arecorrectly sized and oriented for further processing. The depth andintensity images 718, 720 can be utilized either in one of or both oftwo pathways to detect pavement features—i.e. the depth and intensityimages 718, 720 can be processed by traditional computer visiontechniques 724 and/or can be fed to a machine learning algorithm 726.

With computer vision algorithms 709, the system and method can detectpavement features using deterministic analysis. These computer visionalgorithms 724 try to capture all pavement distresses visible in theinput data, and are used to detect a set of pavement features. Thisalgorithmic process is achieved through amplifying features using acombination of filtering techniques such as Gaussian filter, Gaborfilter, Thresholding, and Laplacian filter. This is followed by edgedetection of Harris corners to isolate the features, and then a localbinary pattern operator is used. All of these filtering methods have incommon an objective to highlight the photometric and geometric aspect ofthe distress.

These computer vision algorithms 724 may detect distresses such as, butnot limited to, cracks, potholes, rutting and transverse profiling,bleeding, and patching. As these distresses are detected, the system andmethod geo-references them in the survey scan data such that theirlocation is accurately recorded. As some distress features are moresuitable to be detected and extracted from the intensity image and somefeature are extracted from the range image, the system and method may beconfigured to prefer one image over another for identifying differentfeatures. The results are combined together and used to categorize thedistress features into one of a number of classes 604-608.

Still referring to FIG. 7, a machine learning module 726 is adapted toperform pavement distress detection in conjuction with, or in additionto, the computer vision algorithms 724. In an embodiment, the machinelearning module 726 comprises an artificial intelligence (AI) engineexecuting a learning algorithm to detect and classify distresses inimages received from the image processing block 722. This process allowsfor learning through feedback to automatic improvement of pavementfeature detection by comparing calculated detection results withpositive identification of the actual distresses.

In an embodiment, the machine learning module 726 can be configured toprovide feedback in real time to change a parameter of a component onthe survey vehicle, in order to address lack of image quality, or tooptimize image collection given changing survey conditions and theoperating environment. For example, the machine learning module feedbacksignal can be processed to change a parameter of one or more lightingmodules to achieve better contrast. As another example, a parameter onthe one or more stereoscopic cameras may be changed to obtain betterquality images for detecting a particular type of pavement distress. Asthe machine learning module 726 can learn over time which parameterswould optimize image capture and processing for a given type of pavementdistress in a given survey environment, accurate pavement distressdetection performance should improve over time.

In an embodiment, the learning algorithm may be supervised to leveragethe most advanced features of computer vision and machine learning. Forexample, the algorithm can learn from a set of input data that has beenlabeled to build a new generalized model which captures the patternsinside the image data that describes a distress.

The distress detection process can be framed as either an objectdetection task or an instance segmentation task. In an object detectionapproach, the goal is to place a tight-fitting bounding box around eachdefect in the image. In an image segmentation approach, the problem isessentially one of pixel classification, where the goal is to classifyeach image pixel as a defect or not. Instance segmentation is a moredifficult variant of image segmentation, where each segmented pixel mustbe assigned to a particular casting defect.

Many state-of-the-art object detection systems can be used such asregion-based convolution neural network (R-CNN), which creates boundingboxes, or region proposals, using a process called selective search. Ata high level, selective search looks at the image through windows ofdifferent sizes and, for each size, tries to group together adjacentpixels by texture, color, or intensity to identify objects. Once theproposals are created, R-CNN warps the region to a standard square sizeand passes it through a feature extractor. A support vector machine(SVM) classifier is then used to predict what object is present in theimage, if any. Using a different approach, such as region-based fullyconvolutional networks (R-FCN), each component of the object detectionnetwork is replaced by a deep neural network.

In an embodiment, part of the distress detection algorithm is based onthe mask region-based CNN (Mask R-CNN) architecture. This architecturesimultaneously performs object detection and instance segmentation,making it useful for a range of automated inspection tasks.

The advantage of using convolution neural networks (CNN) is that, overtime, the system is learning the intrinsic representation of the inputdata given. The features are then extracted automatically and identifiedby the neural network. In a CNN, pixels from each image are converted toa featurized representation through a series of mathematical operations.The input sequentially goes through a number of processing steps,commonly referred to as layers. By combining multiple layers, it ispossible to develop a complex nonlinear function which can maphigh-dimensional data (such as images) to useful outputs (such asclassification labels). Deep neural networks are, by design,parameterized non-linear functions.

Depending on the type of distresses, a different Machine learningalgorithm may be applied. Binary trees type of algorithms, such asAdaBoost or Random Forest are also used in order to performclassification.

The results of the learning process will be a trained machine learningmodel that can be executed on an input image set to discover a detaileddescriptions of the targeted type of pavement distress. The images usedin this process, depending of the type of distress, are both intensityimages and depth images. The algorithm is thus learning the structure ofthe pavement distress. A trained machine learning model can detect andgeo-reference distresses such as, but not limited to, cracks, potholes,rutting and transverse profiling, bleeding, and patching. This detectiondataset is then used in an “online learning process” to continuouslyimprove the accuracy of the model.

With dedicated hardware, the system and method can survey and detectpavement distresses in real time, and with continuous training, thesystem and method can improve the model constantly, which can becomevery powerful over time. With this process, the model learns in asequential manner, and can adapt locally to survey conditions and thetype of pavement surface being surveyed, such that the detetionalgorithm will be influenced more by recent observations in similarconditions, than by older observations or observations fromsignificantly different surfaces. Optionally, in order to be sure thatthe learning algorithm is making the right choice, a review process canbe performed to determine that the detection of the pavement distressesis accurate.

In an embodiment, depending on the type of distresses to be identified,the algorithm can use computer vision matched filtering techniques. Eachimage pixel is classified as a defect or treated as not being a defect,depending on the features that are computed from a local neighborhoodaround the pixel. Common features include, for example, statisticaldescriptors (mean, standard deviation, skewness, kurtosis, localizedwavelet decomposition, Taylor expansions, Bezier fits or any other typeof polynomial parameterization.

In an embodiment, the algorithm is adapted to used a combination ofcomputer vision and machine learning, called learning feed-forward, aslabeled in FIG. 7. A computer vision algorithm is used in a“pre-processing” phase in the raw image (first part), and the output ofthis first step will be used as input for the machine learningalgorithm. This enables the computer vision algorithm to first detectand classify a pavement distress, and allow the machine learningalgorithm to perform a verification step though its ever-increasingknowledge database, either agreeing with the computer vision algorithm,or possibly coming to different conclusions, requiring further analysisto determine which is correct.

In another embodiment, a learning feedback loop is implemented as a corepart of the machine learning module 726. Acquired images of the surveyedpavement surface are used to constantly keep training the machine toimprove, based on newly collected images of the pavement.Advantageously, the model used by the machine learning algorithm isupdated consistently, and the inference detection uses the most recentmodel. By processing the analysis in realtime on board a vehicle, thelearning feedback can also provide feedback to the system to makeadjustments for capturing images, or for adjusting parameters to obtainbetter images for more accurate detection of different types of pavementdistresses.

Once the pavement has been classified through one or both techniques,the information is stored to a database at 728 for retrieval and furtheranalysis as may be required, whether on board the vehicle, ortransmitted by use of a wireless communication module (e.g. wirelessmodule 530 of FIG. 3) to a remote location.

Illustrative Use Cases

A number of illustrative use cases will now be described to provideexamples of how the present system and method may be used in practice todetect pavement distresses.

Cracks: The input to the machine learning algorithm is the depth imageand the accuracy of the training is highly dependent on the quality ofthe depth image. The quality of depth image may vary depending onseveral external factors such as bad lighting, lose calibration orextreme weather conditions. The system is designed to overcome thesechallenges by constantly training and readjusting of the weights everytime a new dataset is collected. Through the system, cracks could befurther classified into regular, sealed, longitudinal, transverse andalligator cracks. Further post processing like finding connected pixelcomponents and contour methods is applied to classify cracks todifferent categories.

Potholes: The system and method uses the depth information of pavementas a feature to identify the potholes on road. What really separatespotholes from cracks is the surface area and the depth is large comparedto the cracks. The system and method uses initial image filtering as apreprocessing step to eliminate the noise component in the depth image.The next step would be to use image processing techniques like dilationto sperate the pixels values belonging to the potholes. The system andmethod applies a generalized threshold value to mask out pot holes inthe binary image. Further post-processing is applied to classify thepotholes as low, medium and high severities based on the diameter.

Rutting and Transverse Profiling: Rutting is the depression left on roadin the wheel path and to measure this—the system and method uses thedepth image. Preprocessing is done on depth image such as noisefiltering. Rut is identified by finding the depth pixels in wheel pathwithin a certain threshold. The minimum of 12 points or depth connectedpixels are required to define a rut. Transverse profiling is the measureof unevenness on the pavement surface calculated similar to rut.

Bleeding: The main characteristic of bleeding is the tacky shiny surfaceon the road due to accumulation of liquid bituminous material. This isidentifiable by processing the intensity image. As a preprocessing step,the system and method applies a Gaussian filter to spread the intensityevenly across the image and then the system and method takes thehistogram of pixels spread across the image and choose a threshold tofilter out the dark and light pixels connected over a certain length.

Patching: The system and method uses depth images to identify thepatching on pavement which is area of surface that have been removed andreplaced or where additional material has been placed to cover crackingor other distress. The main feature used here is the depth value overthe certain area of the image is nearly the same than rest of the image.Preprocessing step such as filtering is applied to remove the noise.Also, the contrast in color change will be used as a feature inintensity image as the patch could be added to an old pavement with newmaterial. The system and process can combine the depth value of 3D imageand color value of intensity image to identify the patch.

Thus, in an aspect, there is provided a mobile pavement surface scanningsystem for detecting pavement distress, comprising: one or more lightsources mounted to a mobile vehicle for illuminating a pavement surface;one or more stereoscopic image capturing devices mounted to the mobilevehicle for capturing sequential images of the illuminated pavementsurface; a plurality of positioning sensors mounted to the mobilevehicle, the positioning sensors adapted to encode movement of themobile vehicle and provide a synchronization signal for the sequentialimages captured by the one or more stereoscopic image capture devices;and one or more computer processors configured to: synchronize thesequential images captured by each camera of the one or morestereoscopic image capturing devices; generate intensity image pairsfrom the synchronized sequential images; perform a 3D reconstruction ofthe illuminated pavement surface from the intensity image pairs usingstereoscopic principles; generate a depth image and an intensity imagepair from the 3D reconstruction; and process at least one of the depthimage and the intensity image utilizing one or more distress detectionmodules to detect a type of pavement distress.

In an embodiment, the one or more distress detection modules comprise acomputer vision module for detecting pavement distress utilizing atleast one of the depth image and an intensity image pair.

In another embodiment, the one or more distress detection modulesfurther comprise a machine learning module, and the computer visionmodule is adapted to generate a learning feed forward to the machinelearning module.

In another embodiment, the one or more distress detection modulescomprise a machine learning module for detecting pavement distressutilizing at least one of the depth image and an intensity image pair.

In another embodiment, the machine learning module includes a learningfeedback loop to enable the machine learing module to improve detectionof pavement distresses.

In another embodiment, the machine learning module comprises anartificial intelligence (AI) engine executing a learning algorithm todetect and classify distresses based on its iterative training.

In another embodiment, the machine learning module is adapted to providea feedback signal to dynamically change a parameter of a component onthe mobile vehicle for capturing the sequential images on theilluminated pavement surface.

In another embodiment, the feedback signal is processed to change aparameter of the one or more light sources mounted to the mobilevehicle.

In another embodiment, the feedback signal is processed to change aparameter of the one or more stereoscopic image capturing devices.

In another embodiment, the machine learning module is adapted to selecta type of image processing filter in dependence upon the type ofpavement distress being detected.

In another embodiment, the machine learning module is adapted tocategorize the type of pavement distress detected, and to store thegeo-reference for the detected pavement distress in real time as thesurvey data is stored.

In another aspect, there is provided a method of scanning a pavementsurface for detecting pavement distress, comprising: providing one ormore light sources mounted to a mobile vehicle for illuminating apavement surface; providing one or more stereoscopic image capturingdevices mounted to the mobile vehicle for capturing sequential images ofthe illuminated pavement surface; providing a plurality of positioningsensors mounted to the mobile vehicle, the positioning sensors adaptedto encode movement of the mobile vehicle and provide a synchronizationsignal for the sequential images captured by the one or morestereoscopic image capture devices; and providing one or more computerprocessors configured to: synchronize the sequential images captured byeach camera of the one or more stereoscopic image capturing devices;generate intensity image pairs from the synchronized sequential images;perform a 3D reconstruction of the illuminated pavement surface from theintensity image pairs using stereoscopic principles; generate a depthimage and an intensity image pair from the 3D reconstruction; andprocess at least one of the depth image and the intensity imageutilizing one or more distress detection modules to detect a type ofpavement distress.

In an embodiment, the one or more distress detection modules comprise acomputer vision module for detecting pavement distress utilizing atleast one of the depth image and an intensity image pair.

In another embodiment, the one or more distress detection modulesfurther comprise a machine learning module, and the computer visionmodule is adapted to generate a learning feed forward to the machinelearning module.

In another embodiment, the one or more distress detection modulescomprise a machine learning module for detecting pavement distressutilizing at least one of the depth image and an intensity image pair.

In another embodiment, the machine learning module includes a learningfeedback loop to enable the machine learning module to improve detectionof pavement distresses.

In another embodiment, the machine learning module comprises anartificial intelligence (AI) engine executing a learning algorithm todetect and classify distresses based on its iterative training.

In another embodiment, the machine learning module is adapted to providea feedback signal to dynamically change a parameter of a component onthe mobile vehicle for capturing the sequential images on theilluminated pavement surface.

In another embodiment, the feedback signal is processed to change aparameter of the one or more light sources mounted to the mobilevehicle.

In another embodiment, the feedback signal is processed to change aparameter of the one or more stereoscopic image capturing devices.

In another embodiment, the machine learning module is adapted to selecta type of image processing filter in dependence upon the type ofpavement distress being detected.

In another embodiment, the machine learning module is adapted tocategorize the type of pavement distress detected, and to store thegeo-reference for the detected pavement distress in real time as thesurvey data is stored.

Throughout the description and claims to this specification the word“comprise” and variation of that word such as “comprises” and“comprising” are not intended to exclude other additives, components,integrations or steps. While various illustrative embodiments have beendescribed, it will be appreciated that these embodiments are provided asillustrative examples, and are not meant to limit the scope of theinvention, as defined by the following claims.

1. A mobile pavement surface scanning system for detecting pavementdistress, comprising: one or more light sources mounted to a mobilevehicle for illuminating a pavement surface; one or more stereoscopicimage capturing devices mounted to the mobile vehicle for capturingsequential images of the illuminated pavement surface; a plurality ofpositioning sensors mounted to the mobile vehicle, the positioningsensors adapted to encode movement of the mobile vehicle and provide asynchronization signal for the sequential images captured by the one ormore stereoscopic image capture devices; and one or more computerprocessors configured to: synchronize the sequential images captured byeach camera of the one or more stereoscopic image capturing devices;generate intensity image pairs from the synchronized sequential images;perform a 3D reconstruction of the illuminated pavement surface from theintensity image pairs using stereoscopic principles; generate a depthimage and an intensity image pair from the 3D reconstruction; andprocess at least one of the depth image and the intensity imageutilizing one or more distress detection modules to detect a type ofpavement distress.
 2. The system of claim 1, wherein the one or moredistress detection modules comprise a computer vision module fordetecting pavement distress utilizing at least one of the depth imageand an intensity image pair.
 3. The system of claim 2, wherein the oneor more distress detection modules further comprise a machine learningmodule, and the computer vision module is adapted to generate a learningfeed forward to the machine learning module.
 4. The system of claim 1,wherein the one or more distress detection modules comprise a machinelearning module for detecting pavement distress utilizing at least oneof the depth image and an intensity image pair.
 5. The system of claim4, wherein the machine learning module includes a learning feedback loopto enable the machine learing module to improve detection of pavementdistresses.
 6. The system of claim 5, wherein the machine learningmodule comprises an artificial intelligence (AI) engine executing alearning algorithm to detect and classify distresses based on itsiterative training.
 7. The system of claim 6, wherein the machinelearning module is adapted to provide a feedback signal to dynamicallychange a parameter of a component on the mobile vehicle for capturingthe sequential images on the illuminated pavement surface.
 8. The systemof claim 7, wherein the feedback signal is processed to change aparameter of the one or more light sources mounted to the mobilevehicle.
 9. The system of claim 7, wherein the feedback signal isprocessed to change a parameter of the one or more stereoscopic imagecapturing devices.
 10. The system of claim 6, wherein the machinelearning module is adapted to select a type of image processing filterin dependence upon the type of pavement distress being detected.
 11. Thesystem of claim 10, wherein the machine learning module is adapted tocategorize the type of pavement distress detected, and to store thegeo-reference for the detected pavement distress in real time as thesurvey data is stored.
 12. A method of scanning a pavement surface fordetecting pavement distress, comprising: providing one or more lightsources mounted to a mobile vehicle for illuminating a pavement surface;providing one or more stereoscopic image capturing devices mounted tothe mobile vehicle for capturing sequential images of the illuminatedpavement surface; providing a plurality of positioning sensors mountedto the mobile vehicle, the positioning sensors adapted to encodemovement of the mobile vehicle and provide a synchronization signal forthe sequential images captured by the one or more stereoscopic imagecapture devices; and providing one or more computer processorsconfigured to: synchronize the sequential images captured by each cameraof the one or more stereoscopic image capturing devices; generateintensity image pairs from the synchronized sequential images; perform a3D reconstruction of the illuminated pavement surface from the intensityimage pairs using stereoscopic principles; generate a depth image and anintensity image pair from the 3D reconstruction; and process at leastone of the depth image and the intensity image utilizing one or moredistress detection modules to detect a type of pavement distress. 13.The method of claim 12, wherein the one or more distress detectionmodules comprise a computer vision module for detecting pavementdistress utilizing at least one of the depth image and an intensityimage pair.
 14. The method of claim 13, wherein the one or more distressdetection modules further comprise a machine learning module, and thecomputer vision module is adapted to generate a learning feed forward tothe machine learning module.
 15. The method of claim 12, wherein the oneor more distress detection modules comprise a machine learning modulefor detecting pavement distress utilizing at least one of the depthimage and an intensity image pair.
 16. The method of claim 15, whereinthe machine learning module includes a learning feedback loop to enablethe machine learning module to improve detection of pavement distresses.17. The method of claim 16, wherein the machine learning modulecomprises an artificial intelligence (AI) engine executing a learningalgorithm to detect and classify distresses based on its iterativetraining.
 18. The method of claim 17, wherein the machine learningmodule is adapted to provide a feedback signal to dynamically change aparameter of a component on the mobile vehicle for capturing thesequential images on the illuminated pavement surface.
 19. The method ofclaim 18, wherein the feedback signal is processed to change a parameterof the one or more light sources mounted to the mobile vehicle.
 20. Themethod of claim 18, wherein the feedback signal is processed to change aparameter of the one or more stereoscopic image capturing devices. 21.The method of claim 17, wherein the machine learning module is adaptedto select a type of image processing filter in dependence upon the typeof pavement distress being detected.
 22. The method of claim 21, whereinthe machine learning module is adapted to categorize the type ofpavement distress detected, and to store the geo-reference for thedetected pavement distress in real time as the survey data is stored.